{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Summarizing Text V2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RcdcqBkV0MTU"
      },
      "source": [
        "#Summarizing Text with T5\n",
        "Copyright 2020, Denis Rothman. MIT License. Hugging Face usage example was modified for educational purposes.\n",
        "\n",
        "[Hugging Face Models](https://huggingface.co/transformers/model_doc/t5.html)\n",
        "\n",
        "[Hugging Face Framework Usage](https://huggingface.co/transformers/usage.html)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "06QFZGxsf_KJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "931f8431-e4c2-4144-bad7-60a4df9e6ae8"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.6/dist-packages (4.1.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.6/dist-packages (from transformers) (0.0.43)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: tokenizers==0.9.4 in /usr/local/lib/python3.6/dist-packages (from transformers) (0.9.4)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.8)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.8)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.19.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.0.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3tYFf-cEIkKL",
        "outputId": "f72a822c-f875-40e3-cc3f-eb7b37e5e47b"
      },
      "source": [
        "!pip install sentencepiece==0.1.94"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: sentencepiece==0.1.94 in /usr/local/lib/python3.6/dist-packages (0.1.94)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FEQO4tDl7xH_"
      },
      "source": [
        "display_architecture=True"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8suV48O07TW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0743f9df-e4a8-46c3-9d8c-595698fa78a5"
      },
      "source": [
        "import torch\n",
        "import json \n",
        "from transformers import T5Tokenizer, T5ForConditionalGeneration, T5Config\n",
        "\n",
        "model = T5ForConditionalGeneration.from_pretrained('t5-large')\n",
        "tokenizer = T5Tokenizer.from_pretrained('t5-large')\n",
        "device = torch.device('cpu')"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at t5-large were not used when initializing T5ForConditionalGeneration: ['decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight']\n",
            "- This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q6zHDK7I1GsY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c40c5620-12c6-4a99-9f86-7eef28ec0871"
      },
      "source": [
        "if display_architecture==True:\n",
        " print(model.config)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "T5Config {\n",
            "  \"_name_or_path\": \"t5-large\",\n",
            "  \"architectures\": [\n",
            "    \"T5WithLMHeadModel\"\n",
            "  ],\n",
            "  \"d_ff\": 4096,\n",
            "  \"d_kv\": 64,\n",
            "  \"d_model\": 1024,\n",
            "  \"decoder_start_token_id\": 0,\n",
            "  \"dropout_rate\": 0.1,\n",
            "  \"eos_token_id\": 1,\n",
            "  \"feed_forward_proj\": \"relu\",\n",
            "  \"initializer_factor\": 1.0,\n",
            "  \"is_encoder_decoder\": true,\n",
            "  \"layer_norm_epsilon\": 1e-06,\n",
            "  \"model_type\": \"t5\",\n",
            "  \"n_positions\": 512,\n",
            "  \"num_decoder_layers\": 24,\n",
            "  \"num_heads\": 16,\n",
            "  \"num_layers\": 24,\n",
            "  \"output_past\": true,\n",
            "  \"pad_token_id\": 0,\n",
            "  \"relative_attention_num_buckets\": 32,\n",
            "  \"task_specific_params\": {\n",
            "    \"summarization\": {\n",
            "      \"early_stopping\": true,\n",
            "      \"length_penalty\": 2.0,\n",
            "      \"max_length\": 200,\n",
            "      \"min_length\": 30,\n",
            "      \"no_repeat_ngram_size\": 3,\n",
            "      \"num_beams\": 4,\n",
            "      \"prefix\": \"summarize: \"\n",
            "    },\n",
            "    \"translation_en_to_de\": {\n",
            "      \"early_stopping\": true,\n",
            "      \"max_length\": 300,\n",
            "      \"num_beams\": 4,\n",
            "      \"prefix\": \"translate English to German: \"\n",
            "    },\n",
            "    \"translation_en_to_fr\": {\n",
            "      \"early_stopping\": true,\n",
            "      \"max_length\": 300,\n",
            "      \"num_beams\": 4,\n",
            "      \"prefix\": \"translate English to French: \"\n",
            "    },\n",
            "    \"translation_en_to_ro\": {\n",
            "      \"early_stopping\": true,\n",
            "      \"max_length\": 300,\n",
            "      \"num_beams\": 4,\n",
            "      \"prefix\": \"translate English to Romanian: \"\n",
            "    }\n",
            "  },\n",
            "  \"use_cache\": true,\n",
            "  \"vocab_size\": 32128\n",
            "}\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5LaWN15NPIPC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d746c169-acc4-4e6a-e704-b7c4f635adc2"
      },
      "source": [
        "if(display_architecture==True):\n",
        "  print(model)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "T5ForConditionalGeneration(\n",
            "  (shared): Embedding(32128, 1024)\n",
            "  (encoder): T5Stack(\n",
            "    (embed_tokens): Embedding(32128, 1024)\n",
            "    (block): ModuleList(\n",
            "      (0): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (relative_attention_bias): Embedding(32, 16)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (1): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (2): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (3): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (4): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (5): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (6): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (7): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (8): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (9): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (10): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (11): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (12): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (13): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (14): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (15): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (16): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (17): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (18): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (19): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (20): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (21): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (22): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (23): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (final_layer_norm): T5LayerNorm()\n",
            "    (dropout): Dropout(p=0.1, inplace=False)\n",
            "  )\n",
            "  (decoder): T5Stack(\n",
            "    (embed_tokens): Embedding(32128, 1024)\n",
            "    (block): ModuleList(\n",
            "      (0): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (relative_attention_bias): Embedding(32, 16)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (1): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (2): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (3): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (4): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (5): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (6): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (7): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (8): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (9): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (10): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (11): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (12): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (13): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (14): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (15): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (16): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (17): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (18): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (19): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (20): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (21): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (22): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (23): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (final_layer_norm): T5LayerNorm()\n",
            "    (dropout): Dropout(p=0.1, inplace=False)\n",
            "  )\n",
            "  (lm_head): Linear(in_features=1024, out_features=32128, bias=False)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DS2twf1P1UYI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "29de71ca-6f91-4d94-90f6-215b47468d66"
      },
      "source": [
        "if display_architecture==True:\n",
        "  print(model.encoder)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "T5Stack(\n",
            "  (embed_tokens): Embedding(32128, 1024)\n",
            "  (block): ModuleList(\n",
            "    (0): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (relative_attention_bias): Embedding(32, 16)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (1): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (2): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (3): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (4): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (5): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (6): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (7): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (8): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (9): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (10): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (11): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (12): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (13): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (14): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (15): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (16): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (17): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (18): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (19): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (20): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (21): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (22): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (23): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "  )\n",
            "  (final_layer_norm): T5LayerNorm()\n",
            "  (dropout): Dropout(p=0.1, inplace=False)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCwdhX9U1MA5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2d4dfd6-c313-415b-e997-da0b0bedd7df"
      },
      "source": [
        "if display_architecture==True:\n",
        "  print(model.decoder)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "T5Stack(\n",
            "  (embed_tokens): Embedding(32128, 1024)\n",
            "  (block): ModuleList(\n",
            "    (0): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (relative_attention_bias): Embedding(32, 16)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (1): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (2): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (3): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (4): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (5): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (6): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (7): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (8): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (9): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (10): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (11): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (12): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (13): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (14): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (15): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (16): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (17): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (18): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (19): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (20): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (21): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (22): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (23): T5Block(\n",
            "      (layer): ModuleList(\n",
            "        (0): T5LayerSelfAttention(\n",
            "          (SelfAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (1): T5LayerCrossAttention(\n",
            "          (EncDecAttention): T5Attention(\n",
            "            (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (2): T5LayerFF(\n",
            "          (DenseReluDense): T5DenseReluDense(\n",
            "            (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "            (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (layer_norm): T5LayerNorm()\n",
            "          (dropout): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "  )\n",
            "  (final_layer_norm): T5LayerNorm()\n",
            "  (dropout): Dropout(p=0.1, inplace=False)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GmrCDtcL1hPn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3a8d557d-0e1f-4174-f58e-4e0e708434fe"
      },
      "source": [
        "if display_architecture==True:\n",
        "  print(model.forward)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method T5ForConditionalGeneration.forward of T5ForConditionalGeneration(\n",
            "  (shared): Embedding(32128, 1024)\n",
            "  (encoder): T5Stack(\n",
            "    (embed_tokens): Embedding(32128, 1024)\n",
            "    (block): ModuleList(\n",
            "      (0): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (relative_attention_bias): Embedding(32, 16)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (1): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (2): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (3): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (4): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (5): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (6): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (7): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (8): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (9): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (10): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (11): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (12): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (13): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (14): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (15): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (16): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (17): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (18): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (19): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (20): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (21): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (22): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (23): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (final_layer_norm): T5LayerNorm()\n",
            "    (dropout): Dropout(p=0.1, inplace=False)\n",
            "  )\n",
            "  (decoder): T5Stack(\n",
            "    (embed_tokens): Embedding(32128, 1024)\n",
            "    (block): ModuleList(\n",
            "      (0): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (relative_attention_bias): Embedding(32, 16)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (1): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (2): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (3): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (4): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (5): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (6): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (7): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (8): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (9): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (10): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (11): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (12): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (13): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (14): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (15): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (16): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (17): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (18): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (19): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (20): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (21): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (22): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (23): T5Block(\n",
            "        (layer): ModuleList(\n",
            "          (0): T5LayerSelfAttention(\n",
            "            (SelfAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (1): T5LayerCrossAttention(\n",
            "            (EncDecAttention): T5Attention(\n",
            "              (q): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (k): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (v): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "              (o): Linear(in_features=1024, out_features=1024, bias=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "          (2): T5LayerFF(\n",
            "            (DenseReluDense): T5DenseReluDense(\n",
            "              (wi): Linear(in_features=1024, out_features=4096, bias=False)\n",
            "              (wo): Linear(in_features=4096, out_features=1024, bias=False)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "            (layer_norm): T5LayerNorm()\n",
            "            (dropout): Dropout(p=0.1, inplace=False)\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "    )\n",
            "    (final_layer_norm): T5LayerNorm()\n",
            "    (dropout): Dropout(p=0.1, inplace=False)\n",
            "  )\n",
            "  (lm_head): Linear(in_features=1024, out_features=32128, bias=False)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5KfhCrifP01"
      },
      "source": [
        "\n",
        "def summarize(text,ml):\n",
        "  preprocess_text = text.strip().replace(\"\\n\",\"\")\n",
        "  t5_prepared_Text = \"summarize: \"+preprocess_text\n",
        "  print (\"Preprocessed and prepared text: \\n\", t5_prepared_Text)\n",
        "\n",
        "  tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors=\"pt\").to(device)\n",
        "\n",
        "  # summmarize \n",
        "  summary_ids = model.generate(tokenized_text,\n",
        "                                      num_beams=4,\n",
        "                                      no_repeat_ngram_size=2,\n",
        "                                      min_length=30,\n",
        "                                      max_length=ml,\n",
        "                                      early_stopping=True)\n",
        "\n",
        "  output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)\n",
        "  return output"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vqiTNoDc7pOv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c83c8762-48bd-47fe-ee95-0da2acf99ddc"
      },
      "source": [
        "text=\"\"\"\n",
        "The United States Declaration of Independence was the first Etext\n",
        "released by Project Gutenberg, early in 1971.  The title was stored\n",
        "in an emailed instruction set which required a tape or diskpack be\n",
        "hand mounted for retrieval.  The diskpack was the size of a large\n",
        "cake in a cake carrier, cost $1500, and contained 5 megabytes, of\n",
        "which this file took 1-2%.  Two tape backups were kept plus one on\n",
        "paper tape.  The 10,000 files we hope to have online by the end of\n",
        "2001 should take about 1-2% of a comparably priced drive in 2001.\n",
        "\"\"\"\n",
        "print(\"Number of characters:\",len(text))\n",
        "summary=summarize(text,50)\n",
        "print (\"\\n\\nSummarized text: \\n\",summary)\n"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of characters: 534\n",
            "Preprocessed and prepared text: \n",
            " summarize: The United States Declaration of Independence was the first Etextreleased by Project Gutenberg, early in 1971.  The title was storedin an emailed instruction set which required a tape or diskpack behand mounted for retrieval.  The diskpack was the size of a largecake in a cake carrier, cost $1500, and contained 5 megabytes, ofwhich this file took 1-2%.  Two tape backups were kept plus one onpaper tape.  The 10,000 files we hope to have online by the end of2001 should take about 1-2% of a comparably priced drive in 2001.\n",
            "\n",
            "\n",
            "Summarized text: \n",
            " the united states declaration of independence was the first etext published by project gutenberg, early in 1971. the 10,000 files we hope to have online by the end of2001 should take about 1-2% of a comparably priced drive in\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nvB2NenCBfO4"
      },
      "source": [
        "Summarizing the Bill of Rights, Version 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2321zS1Q3jPX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6c78b355-8d54-4553-b6e7-d43f8d2b770c"
      },
      "source": [
        "#Bill of Rights,V\n",
        "text =\"\"\"\n",
        "No person shall be held to answer for a capital, or otherwise infamous crime,\n",
        "unless on a presentment or indictment of a Grand Jury, except in cases arising\n",
        "in the land or naval forces, or in the Militia, when in actual service\n",
        "in time of War or public danger; nor shall any person be subject for\n",
        "the same offense to be twice put in jeopardy of life or limb;\n",
        "nor shall be compelled in any criminal case to be a witness against himself,\n",
        "nor be deprived of life, liberty, or property, without due process of law;\n",
        "nor shall private property be taken for public use without just compensation.\n",
        "\n",
        "\"\"\"\n",
        "print(\"Number of characters:\",len(text))\n",
        "summary=summarize(text,50)\n",
        "print (\"\\n\\nSummarized text: \\n\",summary)\n",
        " "
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of characters: 591\n",
            "Preprocessed and prepared text: \n",
            " summarize: No person shall be held to answer for a capital, or otherwise infamous crime,unless on a presentment or indictment of a Grand Jury, except in cases arisingin the land or naval forces, or in the Militia, when in actual servicein time of War or public danger; nor shall any person be subject forthe same offense to be twice put in jeopardy of life or limb;nor shall be compelled in any criminal case to be a witness against himself,nor be deprived of life, liberty, or property, without due process of law;nor shall private property be taken for public use without just compensation.\n",
            "\n",
            "\n",
            "Summarized text: \n",
            " no person shall be held to answer for a capital, or otherwise infamous crime, unless ona presentment or indictment ofa Grand Jury. nor shall any person be subject for the same offense to be twice put\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zr2A49TDBkZz"
      },
      "source": [
        "Summarizing the Bill of Rights, Version 2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HWMvLGyahPFP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0354d7f5-ec21-476d-eb93-dd66204c30b3"
      },
      "source": [
        "#Bill of Rights,V\n",
        "text =\"\"\"\n",
        "A person must be indicted by a Grand Jury for a capital or infamous crime.\n",
        "There are excpetions in time of war for a person in the army, navy, or national guard.\n",
        "A person can not be judged twice for the same offense or put in a situation of double jeopardy of life.\n",
        "A person can not be asked to be a witness agains herself or himself.\n",
        "A person cannot be deprived of life, liberty or property without due process of law.\n",
        "A person must be compensated for property taken for public use.\n",
        "\"\"\"\n",
        "print(\"Number of characters:\",len(text))\n",
        "summary=summarize(text,50)\n",
        "print (\"\\n\\nSummarized text: \\n\",summary)\n",
        " "
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of characters: 485\n",
            "Preprocessed and prepared text: \n",
            " summarize: A person must be indicted by a Grand Jury for a capital or infamous crime.There are excpetions in time of war for a person in the army, navy, or national guard.A person can not be judged twice for the same offense or put in a situation of double jeopardy of life.A person can not be asked to be a witness agains herself or himself.A person cannot be deprived of life, liberty or property without due process of law.A person must be compensated for property taken for public use.\n",
            "\n",
            "\n",
            "Summarized text: \n",
            " a person cannot be deprived of life, liberty or property without due process of law.A person must be compensated for property taken for public use.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k_h8oQ55_zr5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4bc5bef5-139a-436f-c820-5a5851cfde36"
      },
      "source": [
        "#Montana Corporate Law\n",
        "#https://corporations.uslegal.com/state-corporation-law/montana-corporation-law/#:~:text=Montana%20Corporation%20Law,carrying%20out%20its%20business%20activities.\n",
        "\n",
        "text =\"\"\"The law regarding corporations prescribes that a corporation can be incorporated in the state of Montana to serve any lawful purpose.  In the state of Montana, a corporation has all the powers of a natural person for carrying out its business activities.  The corporation can sue and be sued in its corporate name.  It has perpetual succession.  The corporation can buy, sell or otherwise acquire an interest in a real or personal property.  It can conduct business, carry on operations, and have offices and exercise the powers in a state, territory or district in possession of the U.S., or in a foreign country.  It can appoint officers and agents of the corporation for various duties and fix their compensation.\n",
        "The name of a corporation must contain the word “corporation” or its abbreviation “corp.”  The name of a corporation should not be deceptively similar to the name of another corporation incorporated in the same state.  It should not be deceptively identical to the fictitious name adopted by a foreign corporation having business transactions in the state.\n",
        "The corporation is formed by one or more natural persons by executing and filing articles of incorporation to the secretary of state of filing.  The qualifications for directors are fixed either by articles of incorporation or bylaws.  The names and addresses of the initial directors and purpose of incorporation should be set forth in the articles of incorporation.  The articles of incorporation should contain the corporate name, the number of shares authorized to issue, a brief statement of the character of business carried out by the corporation, the names and addresses of the directors until successors are elected, and name and addresses of incorporators.  The shareholders have the power to change the size of board of directors.\n",
        "\"\"\"\n",
        "print(\"Number of characters:\",len(text))\n",
        "summary=summarize(text,50)\n",
        "print (\"\\n\\nSummarized text: \\n\",summary)\n",
        " "
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of characters: 1816\n",
            "Preprocessed and prepared text: \n",
            " summarize: The law regarding corporations prescribes that a corporation can be incorporated in the state of Montana to serve any lawful purpose.  In the state of Montana, a corporation has all the powers of a natural person for carrying out its business activities.  The corporation can sue and be sued in its corporate name.  It has perpetual succession.  The corporation can buy, sell or otherwise acquire an interest in a real or personal property.  It can conduct business, carry on operations, and have offices and exercise the powers in a state, territory or district in possession of the U.S., or in a foreign country.  It can appoint officers and agents of the corporation for various duties and fix their compensation.The name of a corporation must contain the word “corporation” or its abbreviation “corp.”  The name of a corporation should not be deceptively similar to the name of another corporation incorporated in the same state.  It should not be deceptively identical to the fictitious name adopted by a foreign corporation having business transactions in the state.The corporation is formed by one or more natural persons by executing and filing articles of incorporation to the secretary of state of filing.  The qualifications for directors are fixed either by articles of incorporation or bylaws.  The names and addresses of the initial directors and purpose of incorporation should be set forth in the articles of incorporation.  The articles of incorporation should contain the corporate name, the number of shares authorized to issue, a brief statement of the character of business carried out by the corporation, the names and addresses of the directors until successors are elected, and name and addresses of incorporators.  The shareholders have the power to change the size of board of directors.\n",
            "\n",
            "\n",
            "Summarized text: \n",
            " a corporation can be incorporated in the state of Montana to serve any lawful purpose. the corporation has perpetual succession and can sue and be sued in its corporate name. it can conduct business, carry on operations, and have offices\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}